1. Field of the Invention
The invention relates to high-speed video event detection, motion analysis, image recording, and automated image analysis.
2. Description of the Related Art
It is well-known in the art to use high-speed image recording devices for motion analysis of mechanical systems that operate too fast for the human eye to see. These devices capture and record hundreds or thousands of images per second of some mechanical process, and then display those images, in slow motion or as still pictures, for human users to see and analyze the high-speed mechanical motions.
Of particular interest is recording rare, short-duration mechanical events that may cause failures in the mechanical process. The fact that these events are both rare and short-duration creates special challenges. Suppose, for example, that the image recording device records 1000 images per second, the event lasts three milliseconds, and occurs on average once an hour. Without some additional mechanism, the human user would need to look at, on average, 3.6 million pictures to find the two or three that contain the event.
It is well-known in the art to address this challenge by providing a trigger signal for the image recording device that indicates when the event has occurred. The image recording device keeps a limited number of the most recent images, say the last one second of recording, and when the trigger signal indicates that the event has occurred, records for a brief additional time and then stops. This gives the user a relatively small number of images to look at both before and after the event. Furthermore, the user knows exactly when each image was captured relative to the time of the event as indicated by the trigger signal.
Clearly, the success of this method depends on being able to generate a suitable trigger signal. It is well-known in the art to use a photodetector for this purpose. A typical photodetector has a light source and a single photoelectric sensor that responds to the intensity of light that is reflected by a point on the surface of an object, or transmitted along a path that an object may cross. A user-adjustable sensitivity threshold establishes a light intensity above which (or below which) an output signal of the photodetector will be energized.
It is often the case that multiple photodetectors are needed to provide the trigger signal. For example, if the mechanical process is a manufacturing line producing discrete objects, and the event corresponds to the production of an object with a missing component, then at least two photodetectors are needed: one to detect that an object is present, and the other to detect the missing component. Sometimes even more than two are needed to detect complex events.
Using photodetectors to provide a trigger signal has some limitations, however, including                a simple measure of the intensity of light transmitted or reflected by one or more points may be insufficient for detecting the event;        it can be difficult to adjust the position of each photodetector so that it is looking at the exact right point;        the points to be measured must not move around during normal operation of the mechanical process; and        the need for multiple photodetectors can make installation and setup difficult.        
It is also known in the art to use a machine vision system to provide a trigger signal. A machine vision system is a device that can capture a digital image of a two-dimensional field of view, and then analyze the image and make decisions. The image is captured by exposing a two-dimensional array of photosensitive elements for a brief period, called the integration or shutter time, to light that has been focused on the array by a lens. The array is called an imager and the individual elements are called pixels. Each pixel measures the intensity of light falling on it during the shutter time. The measured intensity values are then converted to digital numbers and stored in the memory of the vision system to form the image, which is analyzed by a digital processing element such as a computer, using methods well-known in the art to make decisions.
A machine vision system can avoid the limitations of photodetectors. One machine vision system can replace many photodetectors and make sophisticated measurements of extended brightness patterns, instead of just single-point intensity measurements. Adjusting the positions looked at can be done using a graphical user interface instead of a screwdriver and wrench, and those positions can be relocated for each image based on the content of the image itself.
A machine vision system has its own limitations, however, including:                machine vision systems are generally only suitable when the event relates to the inspection of discrete objects; and        machine vision systems are generally too slow to detect short-duration events, and must instead look for some long-duration condition caused by that event, such as a defective product.        
Note that when used to provide a trigger signal, a machine vision system is separate from the high-speed image recording device. It does not see and cannot analyze the images captured by that device, the very images that contain the event that is to be detected. Even if those images could be made available to a machine vision system, they are produced at far too high a rate to be analyzed by machine vision systems of conventional design.
The Vision Detector Method and Apparatus teaches novel methods and systems that can overcome the above-described limitations of prior art photodetectors and machine vision systems for detecting that a triggering event has occurred. These teachings also provide fertile ground for innovation leading to improvements beyond the scope of the original teachings. In the following section the Vision Detector Method and Apparatus is briefly summarized, and a subsequent section lays out the problems to be addressed by the present invention.
Vision Detector Method and Apparatus
The Vision Detector Method and Apparatus provides systems and methods for automatic optoelectronic detection and inspection of objects, based on capturing digital images of a two-dimensional field of view in which an object to be detected or inspected may be located, and then analyzing the images and making decisions. These systems and methods analyze patterns of brightness reflected from extended areas, handle many distinct features on the object, accommodate line changeovers through software means, and handle uncertain and variable object locations. They are less expensive and easier to set up than prior art machine vision systems, and operate at much higher speeds. These systems and methods furthermore make use of multiple perspectives of moving objects, operate without triggers, provide appropriately synchronized output signals, and provide other significant and useful capabilities that will be apparent to those skilled in the art.
One aspect of the Vision Detector Method and Apparatus is an apparatus, called a vision detector, that can capture and analyze a sequence of images at higher speeds than prior art vision systems. An image in such a sequence that is captured and analyzed is called a frame. The rate at which frames are captured and analyzed, called the frame rate, is sufficiently high that a moving object is seen in multiple consecutive frames as it passes through the field of view (FOV). Since the objects moves somewhat between successive frames, it is located in multiple positions in the FOV, and therefore it is seen from multiple viewing perspectives and positions relative to the illumination.
Another aspect of the Vision Detector Method and Apparatus is a method, called dynamic image analysis, for inspecting objects by capturing and analyzing multiple frames for which the object is located in the field of view, and basing a result on a combination of evidence obtained from each of those frames. The method provides significant advantages over prior art machine vision systems that make decisions based on a single frame.
Yet another aspect of the Vision Detector Method and Apparatus is a method, called visual event detection, for detecting events that may occur in the field of view. An event can be an object passing through the field of view, and by using visual event detection the object can be detected without the need for a trigger signal.
Additional aspects of the Vision Detector Method and Apparatus will be apparent by a study of the figures and detailed descriptions given therein.
In order to obtain images from multiple perspectives, it is desirable that an object to be detected or inspected moves no more than a small fraction of the field of view between successive frames, often no more than a few pixels. According to the Vision Detector Method and Apparatus, it is generally desirable that the object motion be no more than about one-quarter of the FOV per frame, and in typical embodiments no more than 5% or less of the FOV. It is desirable that this be achieved not by slowing down a manufacturing process but by providing a sufficiently high frame rate. In an example system the frame rate is at least 200 frames/second, and in another example the frame rate is at least 40 times the average rate at which objects are presented to the vision detector.
An exemplary system is taught that can capture and analyze up to 500 frames/second. This system makes use of an ultra-sensitive imager that has far fewer pixels than prior art vision systems. The high sensitivity allows very short shutter times using very inexpensive LED illumination, which in combination with the relatively small number of pixels allows very short image capture times. The imager is interfaced to a digital signal processor (DSP) that can receive and store pixel data simultaneously with analysis operations. Using methods taught therein and implemented by means of suitable software for the DSP, the time to analyze each frame generally can be kept to within the time needed to capture the next frame. The capture and analysis methods and apparatus combine to provide the desired high frame rate. By carefully matching the capabilities of the imager, DSP, and illumination with the objectives of the invention, the exemplary system can be significantly less expensive than prior art machine vision systems.
The method of visual event detection involves capturing a sequence of frames and analyzing each frame to determine evidence that an event is occurring or has occurred. When visual event detection is used to detect objects without the need for a trigger signal, the analysis would determine evidence that an object is located in the field of view.
In an exemplary method the evidence is in the form of a value, called an object detection weight, that indicates a level of confidence that an object is located in the field of view. The value may be a simple yes/no choice that indicates high or low confidence, a number that indicates a range of levels of confidence, or any item of information that conveys evidence. One example of such a number is a so-called fuzzy logic value, further described therein. Note that no machine can make a perfect decision from an image, and so will instead make judgments based on imperfect evidence.
When performing object detection, a test is made for each frame to decide whether the evidence is sufficient that an object is located in the field of view. If a simple yes/no value is used, the evidence may be considered sufficient if the value is “yes”. If a number is used, sufficiency may be determined by comparing the number to a threshold. Frames where the evidence is sufficient are called active frames. Note that what constitutes sufficient evidence is ultimately defined by a human user who configures the vision detector based on an understanding of the specific application at hand. The vision detector automatically applies that definition in making its decisions.
When performing object detection, each object passing through the field of view will produce multiple active frames due to the high frame rate of the vision detector. These frames may not be strictly consecutive, however, because as the object passes through the field of view there may be some viewing perspectives, or other conditions, for which the evidence that the object is located in the field of view is not sufficient. Therefore it is desirable that detection of an object begins when an active frame is found, but does not end until a number of consecutive inactive frames are found. This number can be chosen as appropriate by a user.
Once a set of active frames has been found that may correspond to an object passing through the field of view, it is desirable to perform a further analysis to determine whether an object has indeed been detected. This further analysis may consider some statistics of the active frames, including the number of active frames, the sum of the object detection weights, the average object detection weight, and the like.
The method of dynamic image analysis involves capturing and analyzing multiple frames to inspect an object, where “inspect” means to determine some information about the status of the object. In one example of this method, the status of an object includes whether or not the object satisfies inspection criteria chosen as appropriate by a user.
In some aspects of the Vision Detector Method and Apparatus dynamic image analysis is combined with visual event detection, so that the active frames chosen by the visual event detection method are the ones used by the dynamic image analysis method to inspect the object. In other aspects of the Vision Detector Method and Apparatus, the frames to be used by dynamic image analysis can be captured in response to a trigger signal.
Each such frame is analyzed to determine evidence that the object satisfies the inspection criteria. In one exemplary method, the evidence is in the form of a value, called an object pass score, that indicates a level of confidence that the object satisfies the inspection criteria. As with object detection weights, the value may be a simple yes/no choice that indicates high or low confidence, a number, such as a fuzzy logic value, that indicates a range of levels of confidence, or any item of information that conveys evidence.
The status of the object may be determined from statistics of the object pass scores, such as an average or percentile of the object pass scores. The status may also be determined by weighted statistics, such as a weighted average or weighted percentile, using the object detection weights. Weighted statistics effectively weight evidence more heavily from frames wherein the confidence is higher that the object is actually located in the field of view for that frame.
Evidence for object detection and inspection is obtained by examining a frame for information about one or more visible features of the object. A visible feature is a portion of the object wherein the amount, pattern, or other characteristic of emitted light conveys information about the presence, identity, or status of the object. Light can be emitted by any process or combination of processes, including but not limited to reflection, transmission, or refraction of a source external or internal to the object, or directly from a source internal to the object.
One aspect of the Vision Detector Method and Apparatus is a method for obtaining evidence, including object detection weights and object pass scores, by image analysis operations on one or more regions of interest in each frame for which the evidence is needed. In an example of this method, the image analysis operation computes a measurement based on the pixel values in the region of interest, where the measurement is responsive to some appropriate characteristic of a visible feature of the object. The measurement is converted to a logic value by a threshold operation, and the logic values obtained from the regions of interest are combined to produce the evidence for the frame. The logic values can be binary or fuzzy logic values, with the thresholds and logical combination being binary or fuzzy as appropriate.
For visual event detection, evidence that an object is located in the field of view is effectively defined by the regions of interest, measurements, thresholds, logical combinations, and other parameters further described herein, which are collectively called the configuration of the vision detector and are chosen by a user as appropriate for a given application of the invention. Similarly, the configuration of the vision detector defines what constitutes sufficient evidence.
For dynamic image analysis, evidence that an object satisfies the inspection criteria is also effectively defined by the configuration of the vision detector.
Discussion of the Problem
Given the limitations of photodetectors and machine vision systems in providing triggers for high-speed event detection, motion analysis, and image recording, there is a need for improved methods and systems that avoid the need for a trigger signal by providing high-speed visual event detection and integrating it with high-speed image recording.
The Vision Detector Method and Apparatus teaches novel image analysis methods and systems that provide, among other benefits, high-speed visual event detection, but without teaching any integration with image recording for use in motion analysis. Thus there is a need for improved methods and systems that combine suitable elements and configurations of the Vision Detector Method and Apparatus with suitable image recording and display capabilities to achieve novel and useful methods and systems for automatic visual detection, recording, and retrieval of events.
Furthermore, the Vision Detector Method and Apparatus provides illustrative embodiments of visual event detection that are primarily intended to detect events corresponding to discrete objects passing through the field of view. While it will be clear to one of ordinary skill that these teachings may be used to detect other types of events, improvements not taught therein may also be useful in detecting such events. Thus there is a need to expand the teachings of visual event detection to improve its utility in detecting a variety of events.